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Introduction 


Due to the increased availability of information from the Internet and other electronic 
resources, all types of librarians are expected to possess information and computer skills to 
actively assist patrons in accessing information and in recognizing reliable resources. Since 
Library and Information Science (LIS) programs prepare graduate students, many with 
multidisciplinary backgrounds requiring little or no technology skills, to work in all types of 
libraries, mastery of information and computer skills is a high priority. In order to plan an 
effective curriculum, those engaged in graduate education of future librarians wonder in what 
ways and to what extent students possess information and computer skills when entering and 
exiting an LIS program. 

Literature Review 

One of the many responsibilities library science faculty have in educating future 
librarians is to make them aware of the information literacy standards issued by the American 
Association of School Librarians (AASL) and the Association of College & Research Libraries 
(ACRL) for the K-12 and higher education communities (AASL, 2007; ACRL, 2000). Librarians 
from all types of libraries are called upon to actively assist patrons in mastering the necessary 
skills to access information with efficient and effective search strategies and to recognize reliable 
resources (Battle, 2007; Katz, 2007a; Petruzzi & Burns, 2006). The literature effectively 
summarizes difficulties in information and computer competencies among patrons, frequently 
highlighting college students (Katz, 2007a; Mech, 2006; Snow & Katz, 2009). 

When these findings are reviewed in light of educating future librarians, the information 
and computer competencies of students pursuing a library science degree is an interesting point 
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that emerges for consideration. In the last few years, the Educational Testing Service (ETS) 
released a standardized test designed to assess the information and computer skills of college 
students. Two assessments were offered, the core, directed at high school seniors and first year 
college students, and the advanced, directed at students transferring from a two year institution or 
in their second half of a four year program (Katz, 2007b). Since the literature generally reports 
on the difficulties college students encounter with information and computer literacy and the 
various ways exams were used to gauge the abilities of undergraduate college students (Gilchrist, 
2009), in general, a question arose concerning the abilities of a targeted group of undergraduate 
students, specifically those pursuing library science. Therefore, a preliminary pilot study with 
undergraduate school library students conducted during 2007-2008 in the Library Science 
Department at Clarion University of Pennsylvania revealed that 77.8% of the participants taking 
the core assessment scored at or above 165 (Krueger, 2008), the cut score for minimum 
information and computer literacy skills determined by ETS (Tannenbaum & Katz, 2008). Since 
the graduate program prepares students for work in a variety of libraries with diverse service 
populations, the mastery of these same skills carries equal significance. Therefore, the research 
questions that emerged were: 

• Would graduate students perform above the cut score designated by ETS for the 
advanced exam? 

• Could the exam be used as a screening tool for entering students and an assessment tool 
for exiting students? 

• Would the exam results point to areas for improvement in the curriculum? 
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Methodology 


Originally, the methodology for this study was to be a pre- and post-test design using the 
ETS iSkills™ advanced exam. The researchers were interested in learning if students would 
perform better after a period of instruction in the MSLS program. This design was abandoned 
after the changes with the iSkills™ exam, described in the next section, prevented the second 
testing session. Instead, the researchers statistically analyzed the overall and individual scores 
that were available and obtained additional data by individually interviewing the students who 
did not pass the iCritical Thinking™ exam and by holding a focus group session with all the 
students who participated. A content analysis was performed on the interview transcripts and 
focus session notes for recurring themes and patterns. 

Participants 

Students enrolled in the MSLS face to face program at the authors' university were 
invited to participate during the 2009-2010 academic year. Twenty of these students took the 
advanced iSkills™ exam during the fall semester. When the time came for them to retake the 
exam, ETS had changed the iSkills™ to the iCritical Thinking™ exam and transferred the exam 
to Certiport for administration. Since Certiport required a signed license agreement before any 
testing could be conducted, the University needed time to review the agreement. After 8 months 
of deliberation Certiport and the University agreed on the terms for testing to begin again. 
Unfortunately, the first set of 20 students had already graduated and the researchers had to wait 
for the new group of incoming MSLS graduate students who enrolled in the face to face program 
for the 2010-2011 academic year. Thirteen of these students agreed to participate and took the 
iCritical Thinking™ exam during the fall semester. When ETS notified the researchers that the 
exam was transferring back to ETS for administration at the end of March, 201 1 (K. Costantino, 
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personal communication, February 4, 2011), the researchers halted any further testing due to the 
complications this presented with student schedules and computer readiness. 

Procedures 

The principal researcher applied for a faculty professional development grant through the 
Pennsylvania State System of Higher Education to secure funding for the exams. Once notified 
of the award, the laptop computers were reexamined for the appropriate specification in order to 
run the iSkills™ advanced test even though they were used for the core exam with undergraduate 
students during the previous academic year. Paperwork was prepared and submitted to the 
university Institutional Review Board (IRB) so the study could commence at the beginning of the 

2009- 2010 academic year. IRB approval was obtained and students were asked to participate 
during one of the evening class sessions and given an informed consent. Some professors also 
offered extra credit points to students for their participation. Sessions were scheduled and 
published so students could select times convenient to them. The principal investigator 
administered the exam in the department’s computer lab. Students were also asked to share their 
individual score reports since institutional data only included the overall score in addition to the 
background information students were asked to provide at the beginning of the exam. 

Once iSkills™ became iCritical Thinking™ and the cutover was made to Certiport in the 

2010- 2011 academic year, new software, the IQsystem, had to be installed to administer the new 
exam. New netbook computers also replaced the older laptops from the prior year, so each had to 
be examined for the appropriate settings and configurations before downloading the IQsystem 
software. It was also necessary for the principal investigator to have admin privileges on the 
netbooks in order to run the updates that occurred between each testing session. Once all the 
computers were operational, a testing schedule was created and circulated among the students so 
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they could select convenient times according to their schedules. Students were also instructed to 
register with Certiport in order to create their profile and usernames and passwords before 
actually taking the exam. Extra credit in two classes was again offered for participation. Thirteen 
students participated in this phase of the study and institutional data were accessed in the 
Certiport administrator’s module. This time, though, in addition to the overall score, percentage 
breakdowns for each participant were calculated according to the skill set categories for 
accessing, communicating, creating, defining, evaluating, integrating, and managing information. 
This information was entered into SPSS for statistical analysis. ETS scores were entered 
according to the comparison tables given by Certiport (J. Barlow, personal communication, July, 

2010) in order to use one score scale when completing the statistical analysis. Other variables 
used in SPSS were gender, undergraduate GPA, graduate GPA, years of work experience, 
undergraduate major, and whether English was the student’s native tongue. 

Further testing was curtailed when ETS announced that it would reclaim the exam from 
Certiport at the beginning of April, 2011 (K. Costantino, personal communication, February 4, 

2011) . Since this would involve another platform, new software configurations, and a new 
learning curve for students, time did not permit extending the study any further. The researchers 
decided to interview students who received a score below the 260 cut score and then hold a 
group session with all students to explore their impressions of the exam. Individual students were 
interviewed first and asked the following questions: 

• What portions of the exam did you find difficult? 

• What parts of the exam were easy? 

• What did you think about the exam? 
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How would you describe the exam as a measure of your knowledge for information and 


computer literacy? 

• How would you improve the exam? 

Eight of the thirteen students attended the focus group session. The researchers asked them the 
same questions that were used during the individual interviews. Afterward, interview transcripts 
and focus group notes were analyzed for recurring themes and patterns. 

Data Analysis 

Statistical Analysis 

Participants’ background information 

There were a total of 33 students in this investigation. As noted earlier, 20 of the students 
took the advanced iSkills™ exam and 13 students took the iCritical Thinking™ exam. Among 
the 33 participants, 18 (54.5%) were female and 15 (45.5%) were male. There were only 2 
students (6.1%) who had a native tongue other than English. There were no significant statistical 
differences between these factors and reported scores. 

Background information collected as part of each exam revealed that students varied in 
years of work experience and undergraduate majors. The participants’ working experience 
ranged from none (n=3, 9.1%) to more than 10 years (n=5, 15.2%). At least 20 students (60.6%) 
had 5 or less years of work experience. Although the participants’ undergraduate majors varied 
from arts to business, the majority (72%) held undergraduate degrees in library science (n=9, 
28%), social sciences or history (n=6, 19%), education (n=4, 12.5%), or English language and 
literature (n=4, 12.5%). This coincides with ongoing admissions data collected by the chair as 
students are admitted into the program. English, history, education, and the undergraduate K-12 
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library science degree are the leading fields for those enrolled in the graduate LIS program. 
While there were no significant statistical differences between years of work experience and 
undergraduate degrees, Table 1 demonstrates a tendency for exam scores to decline with an 
increase of work years. This suggests that students with more work experience were older and 
were removed from an academic setting for a longer period of time. 

Table 1 


Work Experience and Reported Scores 




Reported Score 


Work Experience 

n 

M(SD) 

95% Cl 

None 

3 

250.00 (105.830) 

[-12.90,512.90] 

Less than 2 years 

8 

292.50 ( 43.342) 

[256.26, 328.74] 

More than 2 years 

9 

318.89 ( 53.020) 

[278.13,359.64] 

More than 5 years; 
Less than 10 years 

8 

300.00 ( 60.474) 

[249.44, 350.56] 

More than 10 years 

5 

234.00 (132.023) 

[ 70.07, 397.93] 


Note. CI=confidence interval. 

Students also self-reported undergraduate and graduate grade point averages (GPA). 
Twenty-two students (69%) reported that they achieved a GPA equivalent of an A at their 
undergraduate institution while 8 students (25%) reported achieving a B. Only 2 students (6%) 
indicated receiving a GPA equivalent of a C. The majority of participants (n=30, 91%) have a 
graduate GPA of 3.5 or higher. As shown in Table 2, the mean (M=3.7414) and leptokurdic 
distribution of the graduate GPAs are high and tightly distributed around the mean which is 
similar to the median. This occurs because graduate admissions requirements generally require 
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an undergraduate GPA of 3.0 or better. However, there were no statistically significant 
differences found between the undergraduate or graduate GPA with the reported exam score. 
Table 2 

Graduate GPA Descriptive Statistics: 


Statistic 

N 

33 

Mean 

3.7414 

Median 

3.8000 

Mode 

4.00 

Std. Deviation 

.25595 


Test score results 

Overall, the students performed well on the exam. Test scores ranged from 80 to 410 with 
an average score of 288.79 (M=288.79, 5/7=75.447, Mode= 320, 340). Twenty-one students 
scored above the mean but more (n=24, 73%) scored above the Certiport cut score of 260. This 
left 9 students (27%) scoring below the cut score that indicates an intermediate foundational 
level of information and computer skills. As noted in Table 3, a one sample t test revealed 
statistically significant differences between the reported scores and the cut score. 

Table 3 


Comparison of Reported Scores to Cut Score 



Cut Score=260 



Mean Difference t(32) 

95% 

Reported Score 

28.788 2.192* 

[2.04, 55.54] 


Notes: CI=confidence interval 
*p<.05, two-tailed 
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One advantage of the iCritical Thinking™ score reports furnished by Certiport was that 
the report provided a breakdown of the overall score into the main skills categories, that is, 
access, communicate, create, define, evaluate, integrate, and manage. The older iSkills™ score 
reports furnished a verbal commentary on skills students mastered instead of a percentage of 
correct answers. Consequently, reports for 11 of the 13 students who took the iCritical 
Thinking™ exam were generated and analyzed. The two missing reports were never found in the 
Certiport administrator’s module by the researchers and no explanation as to why they were 
missing was offered by Certiport. A correlation analysis was conducted on the skills categories 
and the overall reported score. Table 4 shows that strong, positive correlations to the reported 
score were found to be statistically significant with all skills categories except evaluate. 

Table 4 


Correlations of Skills Categories with Overall Reported Score 




Categories 




RS 

Access 

Communicate Create Define 

Evaluate 

Integrate 

Manage 

RS 

P. Correlation 1 

.821** 

.626* .773** .760** 

.058 

919** 

.952** 


Notes: RS=Reported Score; P=Pearson 
*p<.05, two-tailed. **p<.01, two-tailed 

Interview and Focus Group Session 

While the graduate students who scored below the cut score of 260 were interviewed 
individually, some consistent themes emerged when their transcript underwent content analysis. 
A summary of the themes for each student is presented in the following table. Pseudonyms were 
used to protect student identity and the meaning of the categories is subsequently explained. 
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Table 5 


Themes from Student Interviews 


Themes 

Tom 

Lynn 

Robert 

Jim 

Time 

X 

X 

X 

X 

Location 

X 

X 

X 


English 




X 

Tech. Problems 





Font Size 


X 



Software 

X 

X 

X 


Netbooks 


X 



Purpose 

X 

X 

X 

X 

Critical Thinking 




Measure 

X 

X 

X 


Definition 


X 



Improvement 





Time 

X 

X 

X 

X 


If the group session revealed similar themes and patterns, it is indicated in the category 
summary. Any new categories that emerged from the group session are designated as such and 
are presented after the individual interview themes. 

Time 

All four of the students felt that either more time should be allowed for the exam or that 
time should not be a consideration. Tom indicated that "I think ... for a lot of the questions. ..I just 
needed more time." Robert indicated that 
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...it's sort of panicky to have to try an answer those questions especially with the ... time 
frame that you have to answer it... it makes you a little more nervous and you might not be 
able to answer the questions as clearly as you would be able to if you had as much time 
as you would need to take it. 

Jim, a normative English speaker, stated "...English is not my native language, you know, so it 
took me a long time ... to figure out what the question said, and then, you know, answer." Lynn 
felt that the test was assessing how quickly one could answer questions instead of how critically. 
She suggested a change in the exam name when she stated "Quick Thinking could be the name." 
Likewise, students in the group session could not see the relationship between critical thinking 
and having the test timed. They felt that answering the questions within the designated time did 
not necessarily indicate critical thinking. 

Location 

Three of the four students believed that the department computer lab was not the best 
place to take the exam since the lab serves as a gathering place for the graduate students during 
the day. Lynn explained that 

The lab is a very difficult to concentrate when you are being timed for an exam. It can be 
quiet and it can be noisy, but regardless it can be difficult to concentrate on something 
when you have only a certain amount of time allotted for when there are other people 
around. 

Tom stated that he was "easily distracted" in the lab and Robert emphasized that he needed 
"silence to concentrate." 

English 
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The background information gathered during the registration process records whether 
English was the student's first, or native, language. A total of three students indicated ability in 
another language and Jim was the only one who scored below the designated cut score for the 
exam. He indicated that it took him "a long time ... to figure out what the question said..." 

Technical Problems 

Technical problems fell into three subcategories: font size, software, and netbooks. Lynn 
was the only one who indicated that she had difficulties with the size of the text used by the 
exam and with completing it on the netbooks. She explained that 

It was very hard to read on the small computers, and the font. I could not, I kept trying to 
get the font to enlarge and with my glasses on the small computer and the tiny font it was, 
I still couldn't see it very well. 

Robert did not necessarily have issues with using the netbooks, but mentioned that he never took 
a standardized test on a computer before and "would've preferred a paper test." The more 
consistent difficulty, though, involved the software, or interface, used for the exam. Tom stated 
that he found "the software difficult to use" and that "it takes me awhile ... to learn how to use 
new software. ..and do a good job." Robert remarked that "It's just how it's presented and how it's 
laid out doesn't always click right away and make me think about the correct answer." Lynn 
stated that "I think that the software was difficult to work with a lot of times, like getting used to 
in the beginning ... I think that the software could have been a little bit more straightforward or 
easier to use." 

The other students in the group session echoed these concerns. They felt that the 
netbooks were cumbersome to use and that the system software or platform was rather "clunky." 
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The students indicated they spent a good bit of time trying to figure out the built in applications, 
such as the spreadsheet, email, or presentation slides. Overall, the applications within the exam 
were not a good replication of actual application programs commonly used by students. 

Purpose 

This category refers to the clarity, or purpose, behind the question. Students could not 
associate skills to the questions being asked. Jim reflected that he thought the exam should 
"concentrate on different skills" instead of using "diagrams ... that need lot of time." Tom stated 
that he "had difficulty in distinguishing between what was just part of the scenario and what was 
sort-of at the core of the problem." He kept thinking about and looking for "the purpose of the 
question." Lynn maintained that 

...there were portions of the exam where I don't think that the question was very clear. 
Like with the power points, they would tell you what they were expecting and then they 
would give you slides, but I think it was very subjective. Like you could kind-of arrange 
them in different ways. 

She continued 

But I kept thinking, what are they looking for as an answer, you know is there a specific 
right or wrong? So it was a little more subjective like somebody could say "Oh well this 
is right" and then somebody could say that the other way was right, so. 

Robert was perplexed by some questions because he knew of others and himself that are 

obviously good at those things, but when you put it in a question form that gives you an 
amount of time to answer these questions it's a lot different. So I don't think that it [the 
test] shows that people can't think that way, it's just providing the information in a 
different way I think... 


14 



The group session students also commented that they questioned the purpose of the exam 
questions or tasks. They kept wondering how the question, especially a multiple choice question, 
proved critical thinking. Overall, they felt limited by the choice of answers and that not enough 
options were presented. The exam items requesting the use of spreadsheets were referenced 
considerably by the students in the group. They felt the exam question did not make sense the 
way they were presented and that there were many different ways to find a solution. Instead of 
actually testing knowledge and skills like in other tests, such as the GRE®, they felt that they 
were just "jumping through hoops" that the iCriticcil Thinking™ exam required. 

Critical Thinking 

Lynn particularly struggled with a definition of critical thinking. She summarized her 
thoughts by saying "thinking quickly is not necessarily thinking critically ... it just didn't seem to 
be a critical thinking exam." She further explained 

I don't think if it took someone two hours to complete the exam and they had every 
question correct, I don' see how that couldn't be critical thinking. I mean, they are still 
using the interface, they are still learning it, but you know every question correctly that's 
critical thinking in two hours, that should be taken into consideration. 

Robert had doubts as to the effectiveness of the exam as a measuring tool. He stressed that he 
knew individuals that did not pass the exam but that he considers them 

very knowledgeable about information and ... very computer literate ... So, I think as a 
measuring tool, I wouldn't put a whole lot of merit on it because I think critical thinking 
is a difficult thing to measure or grade on a numeric scale ... but, as a concrete measuring 
tool, I don't think that it's [the exam] is very useful ... I don't think you can really measure 
critical thinking just in a specific time frame. 
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Tom struggled with the real life scenarios and the practical examples used in the exam questions 
as a measure of critical thinking. He felt that critical thinking was a "strength for me which 
wasn't tested at all." He preferred "less practical and more abstract" questions that would employ 
the use of "logic ... problem solving ... and mathematics." He felt that an "analytical essay" 
would have been a more accurate measure of his abilities. 

Improvement 

When asked how they would improve the exam, all agreed that more time was needed. 
Jim stated that "time was not sufficient" and Tom indicated that "more time would be an 
improvement." Lynn emphatically indicated that "the time was definitely not; they did not give 
you enough time for it." Robert recommended one of two changes: either "more time or no 
time." 

While the group session supported the need for more time, the students also felt that 
clearer instructions would give them a better idea of what to expect. The authenticity of the 
questions was another concern. Instead of "making up applications [spreadsheet, email, and 
presentation programs]" authentic questions or situations should be used. Actually using regular 
spreadsheet, email, and presentation programs and software would better demonstrate abilities in 
these areas. Students also commented that the exam was dated and required updating. 

Additional Comments 

Some students in the group session offered some additional thoughts about the exam. 
They felt it was more a measure of computer literacy rather than critical thinking and that it 
would be more valuable in measuring a specific skill set as prescribed by an employer in an 
actual work situation. Others commented that the exam items were not aimed at the college level 
and could probably be answered by middle or high school students instead. 
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Conclusions 


Practical outcomes of this study focus on the testing environment. The netbooks proved 
to be inappropriate delivery method due to the exam interface. A desktop computer seems that it 
would display the exam in a more user-friendly, readable format than the netbooks can. Also, 
while the department computer lab is a graduate student study and work area, it apparently has 
taken on an atmosphere similar to a student lounge. A quieter location is needed so students can 
concentrate as best possible. 

The value of the exam for graduate students is questionable. Approximately three-fourths 
of the students in this study passed the exam on the first attempt. The remaining students 
expressed valid concerns about the importance of time placed on the exam. It is unclear as to 
how critical thinking is measured in a timed response situation. Students expected abstract 
questions similar to the Graduate Record Exam® (GRE) and felt that the real life scenarios 
described in the iCritical Thinking™ questions did not call for critical thinking as did those from 
the GRE. They did not understand the purpose of the iCritical Thinking™ questions and how 
they could measure critical thinking skills. It could be necessary to offer an alternative method 
for measuring information and computer literacy for students who are more deliberate in their 
approach to standardized testing. Perhaps authentic assessment opportunities emphasized by 
Gilchrist (2009) would offer graduate students more meaningful ways to demonstrate their skills. 
Also, the correlation findings with regards to the evaluate category tasks in some ways echo 
those of Snow and Katz (2009) when they attempted to validate exam scores with more 
naturalistic exercises involving the evaluate tasks. They noted that students were able to select 
their own information and computer technology (ICT) within a computer lab to perform the 
naturalistic, authentic exercises created from actual assignments and interviews of students. 
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Since the students were not bound by the exam web browsers and software, they were able to 
apply information and computer technology skills beyond a predetermined, less authentic setting. 
Embedding such opportunities within the curriculum could generate value-added assessment data 
for enhancing the curriculum and programs of study. 

Further Research 

Obviously, the number of participants and the problems created by the change in vendors 
and the exam itself served as limitations to this study. However, since the time limit on the exam 
was a major issue for the students, it would be interesting to see if the students who scored below 
the desired cut score could achieve it if given more time when retaking the exam. It would also 
be interesting to compare results using the ETS platform since this proved less problematic in 
previous research conducted by the principal investigator. Furthermore, given the changes in 
personal computing options that have emerged since ETS first created the exam, computer skills 
may need to be assessed differently and separately from information literacy skills. Building a 
series of tasks using materials found in an actual school or work environment may shed more 
light on an individual student’s initial skill level with regard to emerging computer technologies 
found in libraries. Once this level is established, course assignments integrating naturalistic and 
authentic scenarios can then demonstrate a student's skill level for defining, accessing, 
evaluating, managing, integrating, creating, and communicating information more effectively. 
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